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ABSTRACT 



Efficient simulation is achieved by employing a highly 
efficient ordering of the events to be simulated. Specifi- 
cally, the events to be simulated are grouped into layers 
and the layers are simulated in order. Each of the layers 
consists of events that are either strictly independent of 
the other events in the layer or are dependent of other 
events in the layer but possess a particular attribute. 
That attribute is one that permits the use of an associa- 
tive operator. This operator allows the simulation of N 
events in 0(log N) computation iterations. 

33 Claims, 5 Drawing Sheets 
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UNBOUNDEDLY PARALLEL SIMULATIONS 

BACKGROUND OF THE INVENTION 

This invention relates to discrete events simulation 5 
and, more specifically, to efficient simulation of events 
in a multiprocessor environment 

Simulation of a discrete event system traditionally 
entails simulating events in time order, be ginning at 
some initial simulated time and progressing forward in 10 
time. This approach normally utilizes a global "simu- 
lated time" clock and an event list. Formerly, simula- 
tions were performed on a single computer and events 
were simulated seriatim, progressing forward in simu- 
lated time. To improve simulation speed, it was natural 15 
to look at the use of more than one processor to simulate 
the event list and, indeed, some speed improvement was 
realized by employing more processors even though the 
basic approach remained the same. However, the im- 
provement in simulation time came at the expense of a 20 
communication burden. The processors that did the 
simulating had to communicate at least some of the 
simulation results to other processors, and the commu- 
nication burden grew very quickly as the number of 
processors increased. 25 

Since the physical system that is simulated typically 
comprises a number of "nodes" in which events occur 
(the term "nodes" herein intends to encompass objects, 
stations, locations, etc. that are associated with events), 
the obvious division of labor among the processors was 30 
to assign a group of nodes to each of the cooperating 
processors. Restricting each processor to simulate 
events that occur at certain nodes, however, imposed a 
synchronization requirement A processor could not be 
allowed to simulate events of a certain node until it was 35 
known that no node at any of the other processors 
would send a message that would affect the simulation 
of that certain node (if "back-tracking'* was not to be 
employed). The high communication burden between 
the computers sparked interest in simulation algorithms 40 
that would reduce this burden. One such algorithm is 
described, for example, in U.S. patent application Ser. 
No. 07/114369, titled "Bounded Lag Distributed Dis- 
crete Event Simulation Method and Apparatus" and 
filed on Oct 28, 1987, M.S. Pat No. 4,901,260. 45 

A recent publication by Chandy and Sherman 
("Space-Time and simulation", Proceedings of Distrib- 
uted Simulation 1989 conference of the Society for 
Computer Simulation), provides an overview of various 
simulation techniques applicable to multiprocessor ar- 50 
rangements. It describes an approach where the all of 
the events of all of the nodes at all times are treated as 
an a priori known whole. They depict this "whole" as a 
rectangle where the nodes are marked along the x axis 
and time is marked along the y axis. The events to be 55 
simulated are points within that rectangle or, more 
specifically, points along vertical time lines that are 
associated with the nodes. 

The concept proposed by Chandy and Sherman is to 
divide the rectangle into a chosen number of arbitrary 60 
regions. The regions may divide the rectangle with 
vertical cuts and/or with horizontal cuts. The former 
segregates nodes while the latter segregates time. Each 
region represents a process that is assigned to a proces- 
sor. Presumably, more than one process can be assigned 65 
to a processor. Still, having preselected the regions and 
made the assignments, the problem remains to deter- 
mine which events will occur in which regions. This 
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problem is not trivial. It is particularly difficult when 
horizontal (time) cuts are made to form the regions, 
because the mere knowledge that an event may occur at 
some node is not sufficient In order to know which 
processor is to simulate an event, the event's absolute 
(simulated) time also needs to be known. 

Chandy and Sherman propose a solution to this prob- 
lem. Specifically, they suggest using estimates of the 
behavior of events in each region, simulating the events 
in the regions based on the created estimates, sending 
messages to neighboring regions based on the simulated 
events to correct the assumptions made, and repeating 
the simulations to account for the newly arrived mes- 
sages that correct the original assumptions. This itera- 
tive "relaxation" process is repeated until the system 
reaches equilibrium state at which the messages sent by 
the processors correspond to the messages that are as- 
sumed to be received by processors. 

The drawback in the Chandy and Sherman approach 
is that the initial assumptions made as to the events 
which occur in a region may be completely wrong. 
Since the regions are divided a priori in terms of time 
and nodes rather than by events, three types of errors 
can be made in the assumptions. Errors that relate to 
whether events actually occur in the region, errors that 
relate to the order in which those events occur, and 
errors that relate to the actual time (vis-a-vis the bound- 
aries of the region) in which the events occur. Because 
errors in the assumptions will almost certainly be made, 
and since those errors and the errors they propagate 
must be corrected, there is little incentive to begin with 
any assumptions. Indeed, a close analysis of the Chandy 
and Sherman approach suggests that their iterative "re- 
laxation algorithm" process works no better with some 
assumptions made than with no assumptions made. The 
consequence is that processors which handle regions of 
events far into the future do not perform useful work 
while the relaxation algorithm either establishes the 
proper conditions from no assumptions, or establishes 
the proper conditions by correcting the assumed condi- 
tions. 

Another drawback of the Candy and Sherman paper 
is their concentration on the rectangle as a whole and 
the separation of the rectangle into regions. While they 
offer some interesting insights of the simulation task as 
a whole, they provide no suggestions on what are 
"good" region selections and what processor assign- 
ments provide faster and more efficient simulations. 
Consequently, although speed of simulation is one of 
the primary goals of simulation methods, a person who 
performs simulations using the Chandy and Sherman 
teachings but with some arbitrarily selected regions will 
not be likely to get the desirable effect of a high speed 
of simulations. 

SUMMARY OF THE INVENTION 

Efficient simulation is achieved, in accordance with 
the principles of this invention, by employing a highly 
efficient ordering of the events to be simulated. Specifi- 
cally, the events to be simulated are grouped into layers 
and the layers are simulated in order. Each of the layers 
consists of events that are either strictly independent of 
the other events in the layer or are dependent on other 
events in the layer but possess a particular attribute. 
That attribute is a particular timing relationship which 
relates the times of the events to one another using an 
associative operator. In complex situations, additional 
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dependencies between these times may be accommo- of the job queues in the workstations. It may be also 

dated by iterative application of procedures involving required to compute various statistics, e.g., the mean 

merging or sorting of the events. Use of the associative queue length. 

operator allows the simulation of N events in a smaller FIG. 2 depicts a possible scenario of job arrivals, 

number of computation iterations, such as in OQog N) 5 processing and completions in the system of FIG. 1. 

computation iterations. An operator is associative when The scenario of FIG. 2 was created randomly for the 

the same result is reached whether the operator is ap- illustrative purposes of this disclosure. The horizontal 

plied to a first intermediate result and event C, or to axis in FIG. 2 is devoted to nodes and the vertical axis 

event A and a second intermediate result— where the is devoted to simulated time. This configuration is the 

first intermediate result is obtained by applying the 10 same as the one described by Chandy and Sherman in 

operator to events A and B, and the second intermedi- the aforementioned article. Each vertical line represents 

ate result is obtained by applying the operator to events the time line of one of the nodes. Specifically, line 11 

B and C. One approach for creating simulation layers is represents the time line of node 10, line 21 represents the 

to separate the events by the simulated nodes (rather time line of node 20, line 31 represents the time line of 

than by time segments, as is typically done in prior art 15 node 30, and line 41 represents the time line of node 40. 

simulations). In FIG.2, each staircase-like shape represents a job. The 

_____ __ __ arrow and the lower edge of the shape represents the 

BRIEF DESCRIPTION OF THE DRAWING ^ ^ 4,,^ hor_on Jedgeof the 

FIG. 1 illustrates a simple four-node serial connection shape represents the job's departure. The stacking of the 

of workstations; 20 staircase-like shapes provides a measure of the queues at 

FIG. 2 depicts the simulated job arrival and depar- the nodes. For example, the job arrivals at node 21 

ture times for the FIG. 1 system; correspond to the job departures at node 11; and, be- 

FIG. 3 presents one structural arrangement for imple- tween the arrival of job 16 and the departure of the job 

meriting the simulations of FIG. 2; corresponding to the arrival of job 22 at node 21, the 

FIG. 4 illustrates a four-node serial connection of 25 queue length at node 11 is three jobs, 

workstations with a merging input; Although FIG. 2 shows all of the events in the time- 

FIG. 5 illustrates a simple four-node serial connection space rectangle bounded by the x and y axes and by 

of workstations with feedback; dashed lines 12 and 13, it should be understood that 

FIG. 6 depicts the simulated job arrival and depar- when simulation starts, neither the existence nor the 

ture times for the FIG. 5 system during the second 30 times of occurrence of these events are known. This 

iteration; information is the immediate result of the simulation. 

FIG. 7 depicts the final simulated job arrival and The ultimate answers pertaining to the developed 

departure times for the FIG. 5 system; and queues come from analyzing the simulated events of the 

FIG. 8 presents an example of a somewhat more completed FIG. 2. These ultimate answers are rela- 

complex system and how the node connections affect 35 tively easy to come by once the FIG. 2 events are shnu- 

thc order of simulations. lated and, consequently, it is the efficient and fast simu- 

nFTATT FT> nFSrttTPTION lati0a ° f CVentS *"* invention --Presses- 

Ufci AiLtu u^tKin iuim as indicated above, one approach is to simulate early 

To better understand our invention it is useful to events first because the early events dictate the later 

center the description around a specific example. For 40 events. This is the ''natural" order. In accordance with 

illustrative purposes, it is assumed, that we need to one approach of our invention, however, simulations 

simulate a system of four workstations (nodes) that are are performed in time slices that are explicitly defined, 

connected serially and together perform useful tasks. and the simulations are carried out without the need for 

Depending on the application, a workstation may be a "roll-back" Thus, with reference to FIG. 2, simulations 

computer, a worker in a production line, a robot, etc. In 45 begin by simulating events within time interval 0 to ti, 

this arrangement, "jobs" are applied to the first station where t\ is a preselected simulated time on the time axis 

in the serial connection at random times, and the arriv- of FIG. 2, Setting the simulation "horizon" to ti causes 

ing jobs have a certain mean and variance of arrival the simulation of job arrival events 14, 15, and 16 on line 

times. Each station processes the arriving jobs on a 11, the simulation of job arrival event 22 on line 21, the 

first-come first-served basis as it becomes free to do the 50 simulation of job arrival event 32 on line 31, and no 

processing. The time that is required to complete each events on line 41. Event 14 represents the arrival of job 

arriving job is random. This randomness also has a 0 at node 10, event 15 represents the arrival of job 1 at 

certain mean and variance. When a new job arrives at a node 10, event 16 represents the arrival of job 2 at node 

node before the previous job ended its processing, the 10, event 22 represents the arrival of job 0 at node 20, 

new job is placed in a queue. Once the job is completed, 55 and event 32 represents the arrival of job 0 at node 30. 

it is forwarded to the next node in the serial connection, The job departure times are also events that occur at 

and the oldest job in the queue is taken up for process- nodes 10, 20, 30 and 40. Thereafter, having completed a 

ing. This arrangement is represented by FIG. 1 in the slice the simulation "horizon" is advanced to a later 

form of a directed graph. Nodes 10 through 40 repre- time, such as time ti and events 17, 23, 24, 33, 34, 42, 43, 

sent the station, and the arrows represent the job paths. 60 .and 44, are simulated. The process continues until time 

The statistics of the arriving jobs in the FIG. 1 ar- , t9, whereupon the simulating of all of the FIG. 2 events 

rangement (e.g. the mean and variance of job arrival is completed. This horizontal slicing which results from 
time intervals) are known, as well as the statistics of the the successive selection of <( horizons" forms, in effect, 

time required by the FIG. 1 station to process the arriv- simulation layers that are processed seriatim, 

ing jobs. The challenge is to efficiently deterrnine, 65 In accordance with another approach of our inven- 

through simulation, the times at which jobs arrive and tion, the events to be simulated are also divided into 
depart each workstation, and subsequently, to effi- layers and the layers are simulated seriatim. However, 
ciently develop the histories and other statistics of the the layers in this approach are selected in a very differ- 



02/27/2004, EAST Version: 1.4.1 



5,375,074 

5 6 

eat manner. Specifically, each layer contains the events 

whose simulations depend on no other events, depend /+i ^ (5) 

on events that were simulated in previous layers, or i> /+Jt m Aj n s r + D f n s r 

depend on events in the current layer that belong to 5 J=i + { r =J r=i+i 

class X. Events are in class X when they can be segre- 
gated into groups and ordered so that events in a group S t mcc < 5 ) expresses D,+* in terms of D,-, a 
can be simulated from events in the previous groups ' lear * r ™* * e *? r .f s ***** « ma * bc a 
with the aid of an associative operator. Hie ordering of ^n H^+* which operates on D/todenve the value 
groups can be in a linear sequence 1-»2-k . . 1— 10 l+k> ' " 

as in the simple case considered here, but can also be D/ + *=H* f+ *(Da. (6) 

more complex; e.g., it may constitute a tree. In all cases, 

however, the notion of a "previous" group must be The question is whether the H operator (operating on 
defined. In the case of a linear sequence, the previous l5 tflc variables identified by the subscript and the super- 
group for group i is group j <i. One consequence of this script) can be expressed in term of the composition of H 
attribute (belonging to class X) is that n events can be operators. If so, that would imply that H n ' 4 (D) can be 
simulated in 0(Iog n) computation iterations. When the ? et ™!? by evaluating Ha' 4 (H n 3(D)) or b y evaluat- 
number of the groups within a layer is large, substantial mg + K f?t ^\7^ m °T &te * WC 

simulation speed-up is derived from the 0(log n) attri- ™ DOte ** ***Q*n*<P» 
bute. The number of events in the conventional time 

slice layers is very small, and its parallelism is bounded. » ** ^ 
Not much benefit is derived, therefore, from the 0(log /= jf + 1 Aj r =j Sr + 
n) attribute, even if the events do belong to class X. 25 

Thus, the benefits of our first-mentioned approach de- ^ ✓ v 

rive primarily from a different aspect of our invention. n S,[ 2 AjUs r + n s^Dd 1. 

On the other hand, the number of events ma time slice r=i3+l ^'='1+1 r=j r=n+i J 

along the time line is unbounded, and hence its parallel- 
ism is unbounded. Events along a time line do belong to 30 Combining terms, we get, as expected, 
class X, as demonstrated below. 

With reference to FIG. 2, the following will show /4 ft (8) 

how to compute the sequence of departure events for £ Aj n s r + Ai n s r 

each of the nodes in the system of FIG. 2. Computing 35 + 1 r =J r=fl +1 

the corresponding sequence of arrival events given the 

inter-arrival periods is similar. Merging the arrival and Slmilar1 ^ WW^)) equals 
departure sequences and summing over the merged 

sequence treating arrivals as H-l's and departures as ^ # *4 ^ 
— Ts, procedures that can be done with great efficiency i Aj J^f r + 

on a multiprocessor, determines the queue length his- 
tory shown in the figure. The time at which the 1 th job > 

departs from a node (and arrives at the next node) can n S r \ % A Tls r + n SADd I 

be expressed as 45 r=i2+i '(y=fl + l r~j r r-n+l J 

«*A-i>+* (1) whichalsoeqTiaIs 

where A/ is the time of arrival of job i at the given node, 
and St is the service interval for job i. Equation (1) can JU i4 i4 

also be written as jj w * + ^ r J+f' 



50 „ „ * (10) 



z^n»^4H-4MA-i+^ (2) Thus, equation (3) becomes D'-H/.i'^-i) and equa- 

55 tion (6) can be interpreted as 

and equation (2) can be written as 

DnzAtSi+Di-t^i (3) Bt +l '+l.H/+\ 

where unlike normal notations the or product, oper- 60 where ' denotes function composition. Since function 
ation represents addition, and the M + " operation repre- composition is associative, we can group the functions 
seats the "max" function. Proceeding from here, k advantageous manner. 

In light of the above, layers that correspond to the 
i?/ + jt»^/+*S/ + *+^/ + jk-iSf + toS/ + 4_i+. . . ti 1116 foes of FIG. 2 constitute one valid layering ap- 

-4/ + j5/ + *S/ + fc„iS/ + fc.2. . . 65 preach in accordance with the principles of this inven- 

aS /+a _iS/ + *_2. . . (4) tion. When such layers are selected, the departure times 

of the jobs along a time line can be evaluated iteratively 
Written in closed form, the above becomes as shown in the table below. 
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job initial iteration 1 iteration 2 iteration 3 

1 T(p irQ) T(t) <i) TQ) w(D TQO 

7 Hfi 7 6 H« 7 H5 6 = H 5 7 5 Hs 7 H 3 5 = H3 7 3 H 3 7 H_i 3 = H_i 7 

< Hi 6 5 Hs 6 *^ 5 = H4 6 4 =H2 6 2 H 2 6 H_i 2 = H_i 6 

5 H4 5 4 H4 5 H 3 4 = H3 5 3 H^H^srHi 5 1 H^HUi 1 = H.j 5 

4 H3 4 3 H 3 4 H2 3 = H 2 4 . 2 H 2 4 Ho* = H© 4 0 Ho 4 H- " 

3 H2 3 2 H 2 3 Hi* = Hi 3 1 Hi 3 H_, 1 = H_j 3 -1 H_i 3 

2 Hi 2 1 Hj^Ho 1 = Ho 2 0 H^H., 0 « H-1 1 -1 H_i 2 
1 Ho 1 0 H«lH.i 0 = H_il -1 H-i» -1 H-i* 
0 H-t° -1 K_i° - 1 H_i° —I H— 1° 



The table demonstrates that, when there are 8 jobs, the 
necessary number of iterations required for developing 
all of the job departure tines, when grouped in an ad- 
vantageous manner, is bounded by log28, or 3. At each 
iteration, operators H which determine departure times 
are computed for jobs in the queue based on a previous 
computation. Each iteration also provides a pointer 7r(i) 
for computing the operators at the next iteration. Thus, 
in connection with job 6, for example, the initial value is 
H5 6 and the pointer tr(€) is set to 5. At the first iteration, 
the computation is based on the value in the row of 
event 5, operator H4 6 is computed and the pointer ir(6) 
is set to the current pointer value 7r(5) of event 5; to wit, 
to 4. At the second iteration, operator H2 6 is computed 
based on the value in the row of event 4 and the pointer 
tt(6) is set to the current pointer value 7r(4) of event 4; 
that is, 2. Finally, in the third iteration, operator H_i 6 is 
computed based on the values in the row of event 2» and 
the pointer tt(6) is set to the current pointer value ?r(2) 
of event 2, i.e., — 1. Having the complete set of operators 
H-]°, H-i l , . . . H_i 7 at iteration 3, we can immedi- 
ately compute the set of departure times 
Do=H_i°D_i, . . . I>7=H_i 7 D_i, where D_i is as- 
sumed 0. 

The above table demonstrates that the events of a 
node can be advantageously assigned to one layer. Typ- 
ically, a simulation effort involves simulating a large 
number of events in a node, and the assigning of a large 
number of events to one layer could appear, on first 
blush, to be disadvantageous. Actually, this works to 
the advantage of this invention because additional pro- 
cessing power can always be applied to simulate the 
node events in parallel, and because the class X attribute 
permits the overall processing of a layer to grow only as 
the logarithm of the number of simulated events. In an 
ultimately parallel environment, a processor can be 
devoted to the simulation of each single event in the 
layer (as compared to the prior art devoting of a proces- 
sor to the simulation of a single node). Once a layer has 
been simulated, information is available to simulate the 
next layer. In connection with the FIG. 2 events and a 
layering approach based on vertical strips, the first layer 
would correspond to line 11, the second layer would 
correspond to line 21, and so forth. 

ha connection with the FIG. 2 events and a layering 
approach based on horizontal strips, the first layer 
would correspond to the strip bounded by (0, ti), the 
second layer would correspond to the strip bounded by 
(tj, t2>, and so forth. 

A system for performing the simulations in accor- 
dance with the principles of our invention can have 
various architectures. It can comprise a single computer 
or many computers that are connected in a network that 
permits communication between the computers. FIG. 3 
presents a simple organization of computers 100 
through 107 that are connected to network 110. Con- 



nection network 110 can be a conventional packet trans- 
mission network, a space division switch, or the like. 

Carrying out the simulations for FIG. 2, the events of 
time line 11 are all simulated in the system of FIG. 3 and 
the results are stored in the computers as inputs to the 
simulation of the second layer — the layer of events 
along time line 21. The process repeats until the events 
of time line 41 are simulated. It may be noted in passing 
that, clearly, it is possible for the number of events in a 
layer to far exceed the number of processors. That does 
not present a problem, however, because more than one 
event can be assigned to each processor. The assign- 
ment can be random but, most productively, a number 
of adjacent events can be assigned to each processor. 
Moreover, we need not imagine that each event is as- 
signed to a particular processor, but that the intermedi- 
ate results needed at each iteration are performed coop- 
eratively by the available processors. 

It may also be mentioned in passing that the user may 
not know at what simulated time to terminate the simu- 
lation. The decision to end simulations may depend, in 
fact, on observed results. This situation can be accom- 
modated by selecting large number of events in a slice, 
treating each slice as a simulation task, and simulating 
each task in accordance with the disclosed principles. 
This is an example where the first approach of our 
invention (explicitly defined horizontal slices) can be 
combined with the second approach of our invention 
(vertical slices where the events are in class X). Of 
course, even when not combined with the second ap- 
proach, when employing the first approach of our in- 
vention the total number of processors are assigned to 
the simulation of events in the slice. Thus, in connection 
with FIG, 3 and the events of the first time slice in FIG. 
2, the eight processors are assigned to the 5 events in the 
four nodes. The amount of memory available on the 
machine used to perform the simulation may constrain 
the number of events that can be simulated concur- 
rently. In that case, memory can be reused (e.g., using 
circular buffering) as the simulation proceeds. 

The FIG. 1 system is rather simple in that it contains 
no inputs other than the single input path to node 10. A 
more complex situation is depicted in FIG. 4, where 
another input path is provided to node 40. This addi- 
tional input path, designated by arrow 48, needs to be 
merged with the input path emanating from node 30. 

When the layer that encompasses node 40 is simu- 
lated, the structure of FIG. 4 calls for a merging of the 
events coming to node 40 from the two paths (the path 
of arrow 48 and the path of arrow 49). Merging of these 
events is not difficult because their times o( arrival are 
known when merging is to proceed, and each of the 
input paths supplies a time-ordered, or sorted, list of 
events. Merging of two sorted lists is well known in the 
art as described, for example, in Batcher, "Sorting Net- 
works and Their Applications'*, AFIPS SJCC 32, 1968. 
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A more complex situation results when there is feed- The system of FIG. 5 includes a feedback path from 

back. Feedback occurs when the output of a node is node 40 to node 10, and that creates a situation where 

reapplied to a previous node. Of course, there must be there are no nodes whose events are independent from 

some control in the system to preclude overload and, all other nodes. Still, node 10 is different from nodes 

typically, that involves some test that is performed on 5 20-30 in that node 10 includes an independent input 

the departing jobs to detennine whether they should be from the path of arrow 50. Utilizing the available infor- 
fed back or not. This test may be based on the nature of mation as early as possible reduces the need for later 

the job's output or it may be based on a simple job corrections and, therefore (all things being equal), it still 

independent rule, such as "send each job back for re- makes sense to start the simulations with node 10. Of 

peated processing, but only once". io course, when node 40 also had an independent input, as 

FIG. 5 depicts a structure that is modeled after the in FIG. 5, then node 10 and node 40 would be topo- 

FIG. 1 organization, but with a "once only" feedback graphically equivalent and the simulation could start 

path from node 40 to node 10. More specifically, in the with either of the two nodes. 

FIG. 5 structure each job that receives service at node From the above it appears clear that the initial task in 

10 (line 11 in FIG. 2) terminates or departs from the 15 an event simulation undertaking is to analyze the di- 

system. It does not re-enter node 20. rected graph that represents the system under consider- 

The simulations task of the FIG. 5 arrangement is not ation, and the rate of the flows expected along those 

as straight forward as that of FIG. 1. The problem is paths. The analysis should develop an ordered list of the 

that the departure times of events in node 10 are depen- events to be simulated in such a way that the simulation 

dent on the arrival times of events from node 40, and 20 layers encompass the largest possible number of events, 

those times are not known and cannot be directly simu- When chosing to create layers that correspond to the 

lated with an associative operator. To resolve this di- events of nodes (as described above), the analysis 

lemma, we use a relaxation algorithm that is similar to should develop an ordered list of the nodes to be simu- 

that of Chandy and Sherman. lated. When developing an ordered list of nodes, the top 

We start with the assumption that there is no feed- 25 of the list should include the nodes that are least depen- 
back from node 40 and that the only input to node 10 is dent of all other nodes. This should be followed by the 
from the path of arrow 50. We then compute event nodes that depend thereon, in the order of the nodes 
departure times at node 40 (and the intermediate nodes) having the fewest number of input paths that depend on 
based on those assumptions. Thereafter, we merge the nodes that are below that node (in the ordered list), 
developed departure times of node 40 with the arrow 50 30 FIG. 8 presents an example. Therein, nodes 74 and 76 
input and recompute the nodes* departure times. This are found to have only independent inputs and, accord- 
iterative process is repeated until the developed events ingly, they are selected first. Their relative order is 
at an iteration are the same as the developed events at unimportant Nodes 75 and 73 are found to be depen- 
the previous iteration. dent on nodes 74 and 76 and, therefore, they should be 

FIG. 2 and FIGS. 6 and 7 demonstrate the simulation 35 selected next. However, node 73 is also dependent on 

of events for the FIG. 5 system. FIG. 2 represents the nodes 72 and 71, which as yet are not in the list and 

first iteration, when no feedback is assumed to be pres- hence potentially below node 73 in the list, while node 

ent. FIG. 6 presents the event simulations of the second 75 is dependent on no nodes that are not already in- 

iteration, and FIG. 7 presents the event simulations of eluded in the list. Accordingly, node 75 is selected first, 

the third and final iteration. 40 followed by node 73. At this point the list comprises, in 

The system depicted in FIG. 1 has one very simple order, nodes 74, 76, 75, and 73. Next, node 71 is deter- 

aspect about it, and that is that node 10 is clearly the mined to be the only remaining node that depends on 

node where the simulations should start. No other node any of the nodes included in the list (depends on node 

can receive jobs (directly or indirectly) from a source 73) and, therefore, it is selected next for the list. Lastly, 

other than node 10. That is not to say that the Simula- 45 node 72 is selected because it depends on node 71 (and 

tions task cannot be performed by starting at another because it is the last remaining node to be included in 

node. However, starting at another node would make the list). The above procedure is circumscribed by the 

the task take somewhat longer to complete. For exam- following two procedures: 

pie, one could assume some job entries to node 20, then 1. select the node(s) that depend on no unknown 

simulate nodes 30, 40 and 10, iteratively correct the 50 inputs or, stated differently, that depend on no 

assumptions regarding the inputs to node 20 and the nodes that are not already in the list. Repeat until 

resulting outputs at node 20 until the system reaches the all nodes are exhausted or until no some nodes 

equilibrium. remain but the remaining nodes fail to meet the 

The system of FIG. 4 has two nodes that accept jobs above criteria. In such a case, go to step 2. 

from outside the system (nodes 10 and 40) but here it is 55 2. Of the remaining nodes, from among the nodes that 

not quite clear that node 10 is the preferred starting depend on nodes that are already in the list, select 

node. One could start with node 40 and the inputs arriv- the node that depends on the fewest number of 

ing at the arrow 49 path, while assuming some inputs nodes that are not already in the list If there is 

from the path of arrow 48, proceeding on the basis of more than one such node, select one arbitrarily, 

the known and assumed information, and correcting the 60 Having made the selection, return to step 1. 

assumptions when more information is known. The fact Having created the basic list, the procedure is to 

that assumptions are made and that an iterative "loop" execute the simulations in the list, and to repeat the 

must be engaged to correct for errors in the assumptions simulations in the list, in order, until the equilibrium is 

makes it clear' that a starting node should, if possible, be reached. We found that the needed number of repeti- 

one that does not require any assumptions. Stated differ- 65 tions is very small; usually, on the order of log N, where 

ently, it should comprise events that are least dependent N is the number of events to be simulated, 

of all other nodes. In FIG. 4, the "natural" order 10, 20, Generalizing on the above, any directed graph can be 

30, 40 requires no such assumptions. analyzed to form an ordered list as described above. 
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Creating the list when the graph is acyclic is straight ti ve operator is of order N, so it would take order of log 
forward. When the graph has cycles, however, the N iterations to complete the simulation, 
situation is somewhat different Event situations where different jobs have different 
One approach for dealing with a directed graph that priorities can be handled with this invention. For exam- 
include cycles is to redraw the graph in a form that is 5 pie, when jobs appear with one of two priorities, the 
devoid of cycles. This is accomplished by representing higher priority jobs can be handled first, as if the other 
the cycles encompassed by each strongly connected priority jobs did not exist, and then the lower priority 
component as a single node. A strongly connected com- jobs would be handled. 

ponent is the set of all nodes, and the corresponding Another aspect of job simulations that imparts a "pri- 
links, that are mutually reachable from any of the nodes 1° ority" flavor is related to the physical limitations that 
within the set. Having developed an acyclic representa- the simulation system of FIG. 3 may possess. Specifl- 
tion of the directed given, parent, graph, the ordered list cally, all systems in today's architectures have a mem- 
can be created and simulated. In the course of simulat- ory size limitation, and this memory limitation may 
ing the strongly connected components in the list, the impose an ordering of the simulated events. For exam- 
structure of the strongly connected component is ana- 15 pie, in connection with the FIG. 4 arrangement, if the 
lyzed as an independent graph with the appropriate events arriving at node 10 arrive earlier than the events 
entry points from the parent graph. The event list devel- arriving at node 40 (path 49) and the number of such 
oped for the strongly connected component is simulated events is large enough to pose a memory overflow 
the necessary number of times until equilibrium is concern, it makes sense to devote more resources to 
reached before the next node in the parent list is simu- 20 simulating the events of node 10 than to simulating the 
lated. events at node 40. Because of the complexity inherent in 

The above description presents the principles of this the novel concept of the "associative operator", the 
invention by way of specific techniques and examples, above description concentrates on the 'Vertical" layers 
but it should be appreciated that many variations can be are applicable to other layering approaches including, 
made without departing from the spirit and scope of this specifically, the "horizontal" layering approach de- 
invention. For example, when chosing to develop simu- scribed above. Indeed, it may be pointed out that in 
lation layers by creating vertical strips that encompass some situations the second approach of our invention, 
the events of a node, there is no reason to insist that where "horizontal", or "time , slices are selected in an 
each strip include one and only one node. Indeed, in 30 explicitly defined manner and all of the processing 
connection with FIG. 8, the analysis revealed the inter- power is devoted to simulating the slices to completion 
esting situation that nodes 76 and 74 are interchangeable in a seriatim manner, may be better than the 'Vertical" 
in the ordered list. That implies that nodes 74 and 76 can layering approach. As an example of such a situation, 
be combined into a single Layer. When enough process- one might have (some time in the future) a massively 
ing power is available, such combining can further 35 parallel processor with perhaps 1,000,000 individual 
speed up the simulations. processors working in parallel, for the system to be 

Another interesting situation results when the entire simulated consists of 1,000 nodes. Assigning all 
directed graph is a strongly connected component with- 1,000,000 computing units for processing events in the 
out any entry points. An example of that may be a ring first node, then the second node, in the chosen order is 
network of 5 workstations with 10 tokens constantly 40 clearly possible (in accordance with the above- 
circulating from one workstation to the next. It can be described principles of our second approach) it may be 
shown that the 50 events which represent the set of 10 not advantageous. There simply may not be a need to 
token-processing jobs that pass through the 5 worksta- simulate the system a million events into the future of 
tions can be considered as a group, because an associa- each node. Instead, one might employ the first approach 
rive operator can be found for the resulting groups. This 45 of our invention, define horizontal time slices (perhaps 
50-event group is qualitatively similar to the one-event so that each node has about 1,000 events to process in 
group represented above by D,\ The number of such each slice) and assign the 1,000,000 processors to simu- 
groups will be of order of N where N is the number of late the 1,000 events in each of the 1,000 nodes, 
simulated events, so the available parallelism, as before, We claims: 

is still of order of N. 50 1. A method for simulating on a computer events of a 

Still another interesting situation results when work- system comprising the steps of: 

stations have limited input buffers which prevent the selecting a cluster of events of said system that in- 

nodes from creating boundless job queues. It can be eludes primarily events that are related to each 

shown that in this situation, order of N groups of certain other through an associative operator, an operator 

sets of events may be formed across all nodes, and that 55 being associative when the same result is reached 

an associative operator can be found for these groups. whether the operator is applied to a first intermedi- 

Yet another interesting situation that benefits from ate result and event C, or to event A and a second 

the principles of this invention is found in the ALOHA intermediate result — where the first intermediate 

protocol. In the ALOHA arrangement, a plurality of result is obtained by applying the operator to 

workstations communicate over a common channel, (c events A and B, and the second intermediate result 

When a collision of communication packets occurs, the is obtained by applying the operator to events B 

protocol calls for the colliding workstations to be in- and Q 

formed of the collision. Each of the involved stations simulating the events of said cluster of events; and 

waits a random interval and then attempts to retransmit returning to said step of selecting when at least some 

This situation employs the associative operator to deter- <5 of said events of said system have not been simu- 

mine the simulated departure times, and merging proce- lated. 

dure to reinsert the retransmission events into the events 2. The method of claim 1 wherein said step of select- 
set. The number of groups for application of the associa- ing excludes events that belong to another cluster. 
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3. The method of claim 1 wherein said step of simulat- 
ing, when simulation information about events in other 
clusters is required, utilizes the simulation information 
of said events in other clusters developed by preceding 
steps of simulating, and makes assumptions about the 5 
simulation information of said events in other clusters to 
which said step of simulating was not applied. 

4. The method of claim 3 wherein said steps of select- 
ing, simulating and returning form a sequence of cluster 
simulations, and at least a portion of said sequence is 10 
repeated until the simulation information for all of the 
events in repetition k of said sequence, where k is an 
integer, is the same as the simulation information for all 
of the events in repetition k-1 of said sequence. 

5. A method for simulating on a computer events of a 
system comprising the steps of: 

selecting a layer of events of said system that includes 
mostly event groups that are related to each other 
through an associative operator, an operator being 2 q 
associative when the same result is reached 
whether the operator is applied to a first intermedi- 
ate result and event C, or to event A and a second 
intermediate result — where the first intermediate 
result is obtained by applying the operator to 25 
events A and B, and the second intermediate result 
is obtained by applying the operator to events B 
and C; 

simulating said layer of events; and 
returning to said step of selecting when at least some 30 
of said system events have not been simulated* 

6. The method of claim 5 wherein said step of select- 
ing excludes events that belong to another layer. 

7. The method of claim 5 wherein said step of simulat- 
ing, when simulation information about events in other 35 
clusters is required, utilizes the simulation information 
of said events in other clusters developed by preceding 
steps of simulating, and makes assumptions about the 
simulation information of said events in other clusters to 
which said step of simulating was not applied. 40 

8. The method of claim 5 wherein said steps of select- 
ing, simulating and returning form a sequence of cluster 
simulations, and at least a portion of said sequence is 
repeated until the simulation information for all of the 
events in repetition k of said sequence, where k is an 45 
integer, is the same as the simulation information for all 
of the events in repetition k-1 of said sequence, 

9. The method of claim 5 wherein the event groups 
that are related to each other through an associative 
operator are such that the groups can be ordered and 
the events of a group can be simulated from the events 
of some previous groups with the aid of said associative 
operator. 

10. The method of claim 5 wherein said system com- ^ 
prises a plurality of interacting nodes and each of said 
layers comprises primarily the events of one of said 
nodes. 

11. The method of claim 5 wherein said system com- 
prises a plurality of interacting nodes and each of said ^ 
layers consists the events of one of said nodes. 

12. The method of claim 5 wherein said step of simu- 
lating a layer includes a step of merging the events of 
other layers that affect said layer. 

13. The method of claim 5 wherein said step of simu- 65 
lating a layer includes the steps of 

merging the simulated events of layers that affect said 
layer and that have been previously simulated; and 
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merging assumed events of layers that affect said 
layer but which have not been previously simu- 
lated. 

14. The method of claim 13 wherein said steps of 
selecting, simulating and returning from a sequence of 
cluster simulations, and at least a portion of said se- 
quence is repeated until the simulation information for 
all of the events in repetition k of said sequence, where 
k is an integer, is the same as the simulation information 
for all of the events in repetition k-1 of said sequence. 

15. A method for discrete event simulation on a com- 
puter of system events occurring in a plurality of nodes 
in a multi-node system, where events in one node corre- 
spond to a time interval having more than one time 
sample and affect events in another node, comprising 
the steps of: 

selecting the events of a node; 

simulating events of the selected node; and 

returning to said step of selecting until the last node in 

said order has been simulated and all events have 

been simulated. 

16. The method of claim 15 wherein said simulating 
comprises simulating departure times of jobs. 

17. The method of claim 15 wherein said simulating is 
carried out with the aid of an associative operator. 

18. The method of claim 17 wherein said associative 
operator includes a "max" function which selects the 
larger of said two inputs. 

19. The method of claim 15 wherein said step of se- 
lecting includes a step of formulating a simulations 
order of said nodes. 

20. The method of claim 19 wherein said step of for- 
mulating a simulations order is based on an acyclic 
directed graph representation of said system, which 
graph comprises at least one node from the set of nodes 
that includes strongly connected component nodes and 
non-strongly connected component nodes. 

2L A method for discrete event simulation on a com- 
puter of system events occurring in a plurality of nodes 
in a multi-node system, where events in one node corre- 
spond to a time interval having more than one time 
sample and affect events in another node, comprising 
the steps of: 

formulating a simulations order for simulating said 
nodes; 

designating said the first node in said order as the 
simulation node; 

simulating events scheduled for said simulation node; 

designating a new simulation node by selecting the 
node that follows, in said order, the current simula- 
tion node; and 

returning to said step of simulating until the last node 
in said order has been simulated and all events have 
been simulated. 

22. The method of claim 21 wherein said step of for- 
mulating a simulations order comprises: 

attaining a directed parent graph to represent the 
interaction of said plurality of nodes; 

creating an acyclic directed graph from said directed 
parent graph by representing each strongly con- 
nected component in said parent graph by node; 

selecting a node in said acyclic directed graph that is 
dependent on the fewest number of other nodes in 
said acyclic directed graph; 

assigning the selected node to a list when the selected 
node is nonstrongly connected component node; 
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developing a sublist when the selected node is a 
strongly connected component node and append- 
ing said sublist to said list; and 

returning to said step of selecting until the last node in 
said acyclic directed graph has been selected. 

23. A. method for discrete event simulation on a com- 
puter of system events occurring in a plurality of nodes 
in a multi-node system, where events in one node corre- 
spond to a time interval having more than one time 
sample and affect events in another node and where the 
termination of the appearance on independent events at 
one or more of the system nodes is not known* compris- 
ing the steps of: 

selecting a super-group of events; 
for events within the selected super-group, executing 

a simulation procedure including 

selecting a cluster of events from among the events 
within the selected super-group, which cluster 
includes primarily events that are related to each 
other through an associative operator; 

simulating the events of said cluster of events; and 
returning to said step of selecting when at least 25 
some of said events of said system have not been 
simulated; 

selecting another super-group of events; and 
returning to said step of executing a simulation proce- 
dure. 

24. The method of claim 5 wherein said step of simu- 
lating develops an order of simulating the events in said 
layer. 

25. The method of claim 24 wherein said order is 
related to priority of the events being simulated. 

26. The method of claim 24 wherein said order is 
related to constraints of hardware in which said method 
of simulating is carried out. 40 
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27. The method of claim 24 wherein said order is 
related to memory constraints of hardware in which 
said method of simulating is carried out. 

28. The method of claim 1 wherein said step of select- 
ing selected the events of a cluster based on said associa- 
tive operator. 

29. The method of claim 1 wherein said step of simu- 
lating simulates the events with the aid of said associa- 
tive operator. 

30. The method of claim 1 wherein said step of select- 
ing selected the events of a cluster based on said associa- 
tive operator, and said step of simulating simulates the 
events with the aid of said associative operator. 

31. A method for simulating events of a system with 
an available number of processors comprising the steps 
of: 

dividing the events to be simulated into layers having 
denned interface borders between the layers where 
at least in some of the layers one of the events in a 
layer is causally related to at least one other event 
in the layer; 

selecting an order of simulation for simulating said 
layers; and 

simulating the layers in a seriatim manner, where 
each step of simulating a layer completely simu- 
lates the layer by employing essentially all of the 
available number of processors. 

32. The method of claim 31 wherein said step of di- 
viding the events to be simulated creates clusters of 
events where each cluster of events of said system in- 
cludes primarily events that are related to each other 
through an associative operator. 

33. The method of claim 31 wherein said step of di- 
viding the events to be simulated creates clusters of 
events where each cluster of events of said system in- 
cludes events within a lower border and an upper bor- 
der of the simulated time where the lower border and 
the upper border encompass events of more than one 
particular time. 
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[57] ABSTRACT 

A discrete event simulation system that avoids all block- 
ing and advances the simulation time in an efficient 
manner by treating the simulated system as a set of 
subsystems and simulating the subsystems concurrently. 
The simulation proceeds iteratively by restricting the 
simulation of scheduled events for each subsystem at 
<any~one~time~ to a chosen simulated time segment 
(bounded lag) beginning with the lowest simulation 
time found among the subsystems. With each simulation 
iteration, an "at risk",, demarcation time is evaluated 
based only on a subset of the subsystems that can poteh-v 
dally affect the simulation at the considered subsystem. 
Events scheduled for a time earlier than the "at risk" 
time are simulated. In simulating systems where some 
subsystems affect other subsystems only through inter- 
mediate subsystems, opaque periods can be experienced 
when, because of the specific process that is being simu- 
lated, such an intermediate subsystem "promises" that a 
particular route emanating from this subsystem would 
be busy for a set period of time, and thereby also "prom- 
ises" that no other subsystem can use this route as a 
conduit to affect other subsystems. That tends to push 
forward the "at risk" demarcation time. 
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system as a network of interacting subsystems, distrib- 
BOUNDED LAG DISTRIBUTED DISCRETE uted simulation maps each subsystem onto a processor 

EVENT SIMULATION METHOD AND of the multicomputer network. 

APPARATUS Although distributed simulation provides parallelism 

5 which has the potential for improving the simulation 
BACKGROUND OF THE INVENTION speed, allocation and synchronization of work among 

This invention relates to the simulation art, and more the processors is a major concern which may impede 
particularly to the art of distributed discrete event simu- the realization of the miprovement goals. One well 
J at ^ 0IL known approach for distributed simulation has been 

Computer simulation has become very important in 10 proposed by Chandy and Misra in "Distributed Simula- 
recent years because of the many applications where tion: A Case Study in Design and Verification of Dis- 
simulation of systems is highly beneficial One such tributed Programs," IEEE Transactions on Software 
apphc>atonismeuseofsimulationinthedesignof com- Engineering, Vol. SE-5, No. 5, September 1979, pp. 
plcx systems. These may be electronic systems such as a 440-452, and by Chandy, Holmes and Misra in "Distrib- 
telecommunicatons switching network, robot based 15 uted Simulation of Networks," Computer Networks, 
flexible manufacturing systems, process control sys- Vol. 3, No. 1, February 1979 pp. 105-113. In this ap- 
tems, health care delivery systems, transportation sys- proach, they recognize that physical systems to be simu- 
tems, and the like. Design verification through simula- lated are composed of independent but interacting enti- 
tion plays an improtant role in speeding up the design ties, and that those entities should be mapped onto a 
and insuring that it conforms to the specification. An- 20 topological^ equivalent system of logical nodes. Inter- 
other application is the use of simulation in analyzing, action between nodes is accomplished by the exchange 
and tracking down, faults appearing in operating sys- c f time-stamped messages which include the desired 
tern. Still another application is optimizing the opera- message information and identify the simulation time of 
tion of existing systems through repeated simulations, fh e sending node. In accordancewith the Chandy- 
e.g., the operation of a rnanufecturing facility, the oper- 25 jjolmes-Misra approach, the nodes interact only via 
ation of the telecommunications network, scheduling messages. There are no global shared variables, each 
and dispatching, etc. Yet another application is the use ncK j c ^ activated only in response to a message, each 
of simulation to predict the operation of systems which nDde mnintflim its own clock, and finally, the time- 
for various reasons can not be tested (e.g., resonse to _ stamps 0 f ^ messages generated by each node are 
catastrophe). non-decreasing (in time). In this arrangement, each of 

Simulations can be classified into three types: contra- ^ nodes works independently to process the events 
uous time, discrete time, and discrete event. Discrete ^^ed t0 it ^ ^ correct simulated order. Thus, inde- 
event simulation means simulation of a system in which dent event can ^ simu iated in parallel, within dif- 

phenomena of interest change value or state at discrete ^ ^ if they occur at different simulated 

moments of tune, and no changes occur except m re- 35 ^ 

spoiisetoanar^h^ The time stamping is required, of course, to maintain 

mg a prescribed rou^ defines a discrete event system in ^ ^ f me ^ ge . reccivmg node an event 
which the number of passengers can change only when for ^ x ? not simu £ted when other 

sta^wkth appears that discrete event simulation is less than T. Because of this, when a particular node is 
£SS * ^SSRSiSSm^ because sim- to receive input from two sender ncxies, it cannot 

En oftime when nothing happSs is dispensed with. "ean event with W™*™*^ would^t 
Of course, synchronizaton of the event simulations must *«fed upon to refrain ^^^^ '"g 
be considered when parellelism is employed. Most of- 45 ^ it receives a message f ^^ n ^ ft ^ e 
ten, a discrete event simulator progress by operating Waiting to receive a message 

on an event list An event at the top of the list is pro- simulation process down substantially and can easdy 
cessed, possibly adding events to the list in the course of result m a deadlock cycle where each node waits for a 
processing, and the simulation time is advanced. There- previous node, which amounts to the situation of a node 
after, the processed event at the top of the list is re- 50 waiting for itself. 

moved. This technique limits the speed of simulation to To remedy the wait problem, artisans have been em- 
the rate at which a single processor is able to consider Paying recovery and avoidance techniques. In the re- 
the events one at a time. In a parrellel scheme many covery technique, proposed by Chandy and Misra, 
processors simultaneously are engaged in the task creat- upon detecting a deadlock, the processors in the net- 
ing a potential for speeding up the simulation. Although 55 work exchange messages m order to detenmne which 
techniques for perfonning event list manipulation and of the waiting nodes can process their events in spite of 
event simulation in parallel have been suggested, large the apparent deadlock. This disenbed in K. M. Chandy 
scale performance improvements are achieved only by and J. Misra, "Asynchronous Distributed Simulation 
eliminating the event list in its traditional form. This is via a Sequence of Parallel Computations," Conunumca. 
accomplished by distributed simulations. 60 tions of the ACM, Vol. 24, No. 4> April 1981, pp. 

In a distributed simulation, a number of parallel pro- 198-206. In the avoidance technique, on the other hand, 
cessors form a simulation multicomputer network, and certain types of nodes send null messages under specific 
it is the entire network that is devoted to a simulation conditions even when no instructions for other nodes 
task. More specifically, each processor within the net- are called for. By this technique, nodes can be advanced 
work is devoted to a specific portion of the system that 65 more quickly in their simulation time. Jefferson and 
is simulated; it maintains its own event list and commu- Sowizral, in "Fast Concurrent Simulation Using the 
nicates event occurrences to appropriate neighbor pro- Time Warp Mechanism" Distributed Simulation, 1985, 
cessors. Stated conversely, if one views a simulated The Society for Computer Simulation Multiconference, 
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San Diego, Calif., suggest a different technique where FIG. 3 presents a flow chart describing the steps 
each node is allowed to advance in its simulation time carried out in each of the processors of FIG. 2 in the 
"at its own risk," but when a message arrives that would course of event simulation; and 
have caused some events to not have been simulated, FIG. 4 describes one realization for the node control- 
then a "roll-back** is executed to undo the simulation 5 lers of FIG. 2. 

that was done. Roll-back of a node may not be difficult, ATTcn rwnn-QYtrrinxr 
perhaps, but the fact that the simulated event(s) that DETAILED UfcbCKIrUIUJN 
need to be rolled back may have caused messages to be As indicated above, one of the major problems with 
sent to other nodes does complicate the task substan- the prior art distributed simulation systems is their fail- 
tially. To achieve the rollback, Jefferson et al suggest 10 ure to realize and take advantage of the fact that a delay 
the use of "anti-messages," which are messages that is always present between communicating physical sub- 
parallel the original messages, except that they cause systems. This invention takes advantage of this inherent 
the performance of some action that "undoes" the origi- delay, as described in detail below, 
nal action. FIG. 1 presents a pictorial explanation that may aid in 
Neither of these techniques is very good because each IS understanding the principles of this invention. Therein, 
potentially expands an inordinate amount of computa- vertical lines 21, 22, 23, 24, 25, 26 and 27 represent seven 
tion time in making sure that the overall simulation simulation nodes and their simulation time lines (where 
advances properly. The null message approach expends time advances upward). The circles along these lines 
computing resources in generating, sending, and read- (30-43) represent events that have been, or are sched- 
ing the null message; the recovery approach expends 20 uled to be processed (Le., simulated). These events may 
computing resources to detect and recover from a dead- cause change of value or state, Le., other events, at the 
lock, and roll-back approach expends computing re- originating node or at some other nodes. For purposes 
sources in simulating events and then undoing the work of discussion, it is assumed that node 24 is the node of 
that was previously done. concern, but it is understood that the consideration 

_ ^ w ^^^^^ 25 undertaken by node 24 are concurrently taken by all 

SUMMARY OF THE INVENTION othcr nodcs ^ 

Recognizing that in physical systems there is always The horizontal distances between line 24 and the 
some delay between the time when one part of the other lines represent the time delays for events in other 
system does something, and time when another part of nodes to affect the value or state at node 24. Accord- 
the system realizes that something was done, a Simula- 30 ingly, event 30 processed at node 21 for time Tj may 
tion system is realized that avoids all blocking and ad- cause an event40 at node 24 to be scheduled for some 
vances the simulation time in an efficient manner. The time not earlier than T2, as shown in FIG. 1. The inter- 
efficiency is achieved by each node independently eval- val between Ti and T2 equals the delay between line 24 
uating for itself a time interval that is not "at risk" and and line 21 (Le., the horizontal distance between the 
simulating all events that are within that evaluated time. 35 lines). 

A point in time is not "at risk" for a considered node if The events depicted in FIG. 1 can be divided into 

no other node in the system has a scheduled event that two groups: those that have been simulated (and 

can affect the simulation at the considered node. By also marked with a cross), and those that are yet to be simu- 

restricting the simulation of scheduled events at any one la ted (un-crossed). The simulated events need not be 

time to a chosen simulated time segment (bounded lag) 40 considered for simulation because their effects are al- 

beginning with the lowest simulation time found among ready represented by those events which have not been 

the nodes, allows the evaluation of the "at risk" time simulated (for example, event 30 has caused event 40; 

interval to be based on only a subset of the nodes that the first has been simulated, while the second is yet to be 

can potentially affect the simulation at the considered considered for processing). 

node. This simplification results from the the fact that 45 Of the events that have yet to be simulated (3^-43), 

there are delays between nodes, and that the lower event 33 in FIG. 1 is earliest in time from among all of 

bounds for those delays are fixed and known apriorily. the nodes. In this case, the time of event 33, forms the 

In simulating systems where some nodes affect other current floor simulation time of the system. The floor is 

nodes only through intermediate nodes, opaque periods depicted in FIG. 1 by dashed line 45. In accordance 

can be experienced when, because of the specific pro- 50 with the principles of this invention, a time interval 

cess that is being simulated, such an intermediate node beginning with the floor simulation time is selected for 

"promises" that a particular route emanating from this consideration. This time interval, which I call the 

node would be busy for a set period of time, and thereby bounded lag interval, can be a convenient time interval 

also "promises" that no other node can use this route as that takes into account the number of nodes in the sys- 

a conduit to affect other nodes. That, in effect, increases 55 tern, the number of events to be simulated, and the 

the propagation delay from the nodes that use the busy computing power of the processors employed. AH 

intermediate route, which, in turn, increases the allow* events scheduled within the bounded lag interval can be 

ance for the simulation time within the bounded lag that affected by events scheduled at other nodes within the 

is not "at risk". bounded lag time interval, but only if those nodes are at 

^„^„^^ T A „ ww ^„ inmk1/ , 60 a time distance from the affected node 24 that is less 

BRIEF DESCRIPTION OF THE DRAWING ^ ^ wlectcd ^und* lag interval That reduces the 

FIG. 1 illustrates the timing inter-relationships of number of nodes that need to be considered. In the FIG. 

events processed in a multi-processor environment with 1 depiction, the bounded lag interval ends at dashed line 

recognized delay between events and their effect on 46; and as drawn, the nodes that need to be considered 
other events; 65 for their potential effect on node 24 are nodes 22, 23, 25 

FIG. 2 depicts a block diagram of a multi-processor and 26. Nodes 21 and 27 are outside the bounded lag 

arrangement for simulating events in accordance with delay and their scheduled events within (or outside) the 

the principles of this invention; bounded lag need not be considered. 
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In considering the effects on node 24 in greater detail, 

one can take into account the time of the next scheduled ^ a T{ = ^ 

event and reduce the number of nodes being considered . Sem 71 ^' 
even further. In the FIG. 1 embodiment, for instance, 

the next scheduled event is event 35, and as drawn, only Block 110 can be implemented in each of the controllers 
nodes 23 and 25 need to be considered. by having each controller broadcast to all other con- 
In deteninning whether event 35 is to be simulated, trailers (via network 52) its T/and evaluate for itself the 
one can observe that only event 34 is scheduled early minimum T/ which defines T floor- Alternatively, block 
enough to have a possible impact on event 35. That 10 110 can be implemented within communications and 
event can be analyzed and if it is determined that it does common processing network 52 by having controllers 
not affect event 35, then event 35 can be simulated. 51 send their T/ values to network 52, and having^net- 
Alternatively, it may prove even more efficient to re- work 52 select the minimum T/and broadcast it as Tjioor 
frain from processing event 35 simply because of the back to the contiollexs. 

ftp. this alternative approach is taken because it saves Rollers with which a controller must 
the process of evaluatmg what event 34^may do. coinmum^te, m accordance with block 120, each of the 
FIG. 2 presents a block diagram of a concurrent ^^u^ evaluates its earliest "at risk" time. This is 
event simulator. It comprises a plurality of node con- accomplished by network 52 distributing the T/ infor- 
trollers 51 that are connected to a communications and nation to neighboring controllers, as required, and each 
common processing network 52. Network 52 performs controller Q evaluates the "at risk" demarcation point, 
the communication and synchronization functions, and ttf> f rom th e f t information. This "at risk" point is de- 
in some applications it also performs some of the com- fined as the earliest time at which changes at the neigh- 
putations that are needed by the entire system. The 25 boring controllers can affect the history simulated by 
node controllers can be realized with conventional the controller, based on the neighboring controllers' 
stored program computers that may or may not be own scheduled events or based on a response to an 
identical to each other. Each of the node controllers, event from the controUer itself (reflection). This is ex- 
which corresponds to a node in the FIG. 1 depiction, is pressed by the Mowing equation: 
charged with the responsibility to simulate a preas- 30 
signed subsystem of the simulated system. Each control- 
ler Ci TpflintJtim an event list II* that is executed by a/ = mm[<ty,i) + min{7), d(Uj) + Tf}]. 

simulating each event in the list in strict adherence to J 

me scheduled eventto^ ^ detcnnincdthc va i ue of a, which corresponds 
membered that the bounded lag interval ^selectively to ^ m ^ simu iated time beyond which the 
fixed, and that in conformance with the selected ^ dation of cvcnts at ccmtroUer C, is "at risk", m ac- 
bounded lag interval each controUer 51 is cognizant or cordance ^th block 130, processor d simulates all or 
the processors with which it must interact to determine some c f me scheduled events whose times are earlier 
whether events are scheduled. The process by which ^ ^ a/ ^ 140, me time T/is advanced with each 
the event simulations are enabled is carried out by the simulation of an event, and the simulated event is de- 
controllers in accordance with the above-described leted from II/. Concurrently, if in the course of simulat- 
principles, as shown, for example, by the flowchart of ing an event, new events are called to be scheduled, 
Pig. J, then those events are sent to network 32 for transmis- 
The process described in FIG. 3 is carried out in 45 sion to the appropriate node controllers. Similarly, if 
iterations that encompass blocks lOOthrough 140. Block the execution of events is called to be blocked^ that 
100 is the flow control blocL It tests whether the floor 

simulation time, T^* /« *. * ~r appropnate node controllers for modification of the 
W As longs as ?J^™^Jm^ continues «vatt f<>u carries out ^ cxample depicted in 
by passing control to block 110. When T fl0 or reaches or mQ assuming for the sake of simplifying the draw- 
exceeds T en d t the simulation ends. Block 110 determines ^ ^ situation remains static— i.e., none of the 
the floor simulation time of the simulated system at each depicted events are cancelled and no unshown events 
iteration. That is, block 110 determines the lowest event m scilcdu i ec j t ^ft er evcnt8 30, 31, and 32 have been 
time of me scheduled events aspersed among the event 53 simulated (denoted by the crossed circles) all of the 
lists (II;) of controllers 51 (Q) that are are yet to be no dc controllers communicate their earliest scheduled 
«nymiiqtPH Expressed mathematically, block 110 evalu- event times, T& to network 52 where T/fooris evaluated 
ates the floor simulation time in accordance with the . to correspond to dashed line 45. With reference to node 
equation 24, the bounded lag defined by the distance between 

60 dashed line 45 and dashed line 46 specifies that only 
nodes 22-26 need to be considered. In the course of that 

T floor = min 7* consideration, node 24 determines that scheduled event 

1 S / "a jv 34 a t node 25 defines an "at risk" demarcation point 50. 

Since there are no events scheduled for node controller 

where N is the.total number of node controllers 51 and 65 24 between the time of T/ioor and point 50, no progress 

Ti is the time of the event, et t which has the earliest in simulations is made by this controller. ConcurTendy, 

scheduled time among the events e' in the event list II/; node controUer 22 simulates event 33 (since it is posi- 

■l c tioned at T floor* and no other event can affect it), and 
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node 25 simulates event 34 since neither node 26 nor 
node 24 (the closest nodes) have any events scheduled 
prior to the time of event 34. Node 27 probably also* 
simulates event 36, but this is not certain, since FIG. 1 
does not show all of the neighbors of node controller 
27. 

With events 33, 34, and 36 simulated and deleted from 
their respective event lists, the next iteration raises 
T/joorto the time of event 37 (and correspondingly raises 
the horizon or end of the bounded lag interval. This end 
is equal to Tjioor +B). At this new level of Tjioor the "at 
risk" demarcation point for node 24 is at point 49 (die* 
tated by event 39 of node 23), and in accordance with 
this new "at risk" point, both events 35 and 40 are simu- 
lated within node controller 24. This completes the 15 
simulation of events scheduled for node 24 which are 
shown in FIG. 1. Concurrently at node 23, event 39 is 
simulated, event 37 at node 27 is simulated, but at node 
26 event 38 is not simulated because it is beyond the "at 
risk" point of node 26 caused by the position of event 37 
at node 27. The next T/bor moves to the time of event 38, 
and the process continues to simulate additional events. 

The above description concentrates on evaluating the 
"at risk" demarcation point in connection with the di- 
rect effects of one node on another. In many physical 
systems, however, there are many instances where one 
subsystem affects another subsystem indirectly, i.e., 
through some other subsystem. This condition gives 



20 



25 



^3{ t+1 c» min 

;cneighbors(0 {d(j.t) + max{^/,oi>/J} 
M< 

where op// is the end of opaque period (when communi- 
cation unblocks) for node j in the direction of node i. 

In some simulations it may turn out that the delays 
between subsystems are very small and that opaque 
periods predominate. In such systems the value of each 
at reduces to computing the minimum of the opaque 
periods relative to block i; to wit 



a; = min opjt 
/eneighbon(i) 

m 



It may be observed that with each iteration the value 
of Tjioor increases because the event determining that 
value can always be simulated. The movement of Tjioor 
is affected, however, by how closely the nodes are 
separated and the scheduled events. One other observa- 
tion that can be made is that the above-described proce- 
dure is very conservative, in the sense that an assump- 
tion is made that whenever an event is scheduled to be 
simulated at one node controller, it will always have an 
effect on the neighboring controller (after the appropri- 



rise to the possibility that the mtermediate node is either 3Q ate delay). In physical systems, however, there are 



busy and unavailable generally, or is somehow sensi- 
tized to serve some nodes and not other nodes. Either 
situation can yield the condition that the delay from one 
node, A, to another node, C, through an intermediate 
node, B, is at times much longer than the sum of the of 35 
delays A to B and B to C. I call this additional delay an 
opaque period. Opaque periods have the potential for 
pushing forward the "at risk" demarcation point and, 
therefore, it is beneficial to account for this potential in 
evaluating a/. Such accounting may be achieved by 40 
evaluating a/ iteratively as follows. 



many situations where one part of a system performs 
many operations that affect none of its neighbor subsys- 
tems, or affect only one or very few of its neighbor 
subsystems. Knowledge that an event scheduled for 
simulation will not affect a neighboring node can be put 
to use to simulate more events concurrently (have fewer 
node controllers that are idle). This can be accom- 
plished by communicating not only the T,- of the earliest 
scheduled event in each list, but also the effect that it 
may have on neighboring node controllers. If fact, each 
list can broadcast more than just the earliest scheduled 
event The design decision that a practitioner must 
make, of course, is whether fewer iterations but more 
complex evaluations are economically justifiable. 

FIG. 4 presents a block diagram of one realization for 
node controller 51. Although this embodiment relates 
to use of the iterative method for evaluating a (with the 
use of the auxiliary variable f$) when the delays between 
changes in one subsystem and the effect of those 
changes on other subsystems are not insignificant, it will 
be appreciated by the skilled artisan that the other real- 
izations for computing at are substantially similar to this 
realization. It will also be appreciated that although 
node controller 51 is shown in FIG. 2 as one of a plural- 
ity of controllers, such plurality can be realized within 
a single processor of sufficient computing power. 

In FIG. 4, state register 56 defines the current state of 
the simulated subsystem and event list 53 specifies the 
events that are scheduled to be simulated. Processor 54 
In the above, the term neighbors(i) refers to nodes 60 is the event simulation processor, and it is responsive to 



1: Seta/> = +00 = 7fjJt = 0 
2: synchronize 

3: evaluate /3* +l - min 

yaeighbors<i) + 0/} 
/#/ 

4: evaluate af+ l • min{a/ c ^j c+I } 
5: synchronize 



6: evaluate A = 



45 



50 



55 



I S / S N 



broadcast value of A to all nodes 



that communicate directly with node i. The auxiliary 
variable A* represents an estimate of the earliest time 
when events can affect node i after traversing exactly k 
links. It can be shown that the iteration test is always 
met within a relatively low number of steps, depending 65 
on the value of the bounded lag interval, B. To account 
for opaque periods, the evaluation of £ is augmented to 
be 



state register 56, to event list 53 and to register 58. Reg- 
ister 58 is the a register and, as the name implies, it 
stores the value of a for the controller. Based on the 
value of a and the scheduled time of the event at the top 
of the event list, processor 54 simulates the event in 
conformance with the state of the subsystem and devel- 
ops a new state of the subsystem as well as, perhaps, 
new events. The new state is stored in register 56, new 
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events scheduled for the controller are stored in event 6. The method of claim 4 wherein said subset includes 

list 53 via line 61, and events affecting other controllers the subsystems whose delays are within said preselected 

are communicated out via line 62. Events produced at interval. 

other controllers that may affect this controller are 7. The method of claim 3 wherein said step of evalu- 

accepted via line 63 and stored in event list 53 through 5 ating computes a simulated time value of a and said step 

processor 54. of simulating simulates events whose simulated time is 

Whereas processor 54 is the event simulation proces- not greater than a. 
sor, processor 55 is the synchronization processor. Pro- 8. The method of claim 7 wherein said a for subsys- 
cessor 54 is shown in FIG. 4 as a separate processor but tern i, a,-, is computed by 
in practice a single processor may serve the function of 10 . 
both processor 54 and processor 55. Processor 55 re- 
ceives information from event list 53 concerning the * ± m a + txi 
time of the event in list 53 that possesses the earliest a ' - + - te( * d ™ + r < }] ' 
simulation time. It transmits that information to other 

controllers via line 64 and receives luce information 15 w here dQ4) is said delay between changes occurring at 

from all other relevant nodes via line 65. From that subsystem j and their possible effect on system i,d(ij) is 

information processor 55 develops the value of 1 floor ^ between changes occurring at subsystem i 

and stores that value in register 57, Processor 55 also md thfiir ^ibU effect on system j, and T; is the Simula- 

receives "fit information from its neighbor controllers 2Q don ^ of subsystem j event to be simulated having 

(controllers where changes can affect the controller ^ simulation time from among all other sub- 

directly) via line 66, and transmits its own ft values via sy8tem eyents to ^ simulatcd . 

line 67. With the aid of 1 floor and the other incoming 9 ^ method of claim 4 wnC rein said step of evalu- 

information, processor 55 performs the iterative compu- ^ includes consideration of the condition that the 

tations to develop the values of &*and a t * Those val- 2J effect of a change m one of ^ subsystems is blocked 

ues are stored by processor 55 m registers 57 and 58 t from r ^ through another one of said subsys- 

respectively. terns 

It is to be uiiderstood Oat the foregomg descriptions ■ Qf Qhdm 3 ^ of evalu . 

are merely illustrative of my invention and thatother & ^ n ^ &M ^^ y ^ ofasm ^ dsUp 

implementations and embodiments which incorporate 30 ?^£g simulates events whose simulated time is 

variations from fce above may, nevertheless^ menpo- 0 * fc 5ubs . fc ^ 

rate the : principles thereof For *J» valu f of a{ c m final iteration that follows the steps: 

assigns the task of computing Tjjocr to processor 55. 
However, it may be preferable to include computing 
means in the communication and common processing 33 

network of FIG. 2 where the Tjhor is computed and h Set aP « +« ; fiP = TV; k - 0 

distributed to the various node controllers. 2< gynchroni2e 

I claim: 

1. A method for simulating behavior of a system . ' 

containing interacting subsystems, where a known mini- 40 3: evaluate + = ™\ M 

mum delay exists between changes occurring at one of {WO + h > 

said subsystems and the effects of said changes on others 

of said subsystems, said simulating being performed 4: evaluate af+ l = nunW,0/ +l > 

with a simulation system having a plurality of intercon- 
nected processors with each of said subsystems to be 45 S: synchronize 
simulated being simulated on one of said processors, 
wherein the improvement comprises: 
in each of said processors, a step of simulating events ' cv c 
of subsystems, to be simulated in each of said pro- . 

cessors, whose simulated time Ues within a prese- 50 /*% /' : broadcast vaIuc of A to *" nodcs 

lected interval following the earliest simulation . ~ ' ~ 

time of events yet to be simulated by said proces- 
sor where T/ is the time of the event in subsystem i having 

2. The method of claim 1 wherein each of said steps the earliest simulation time, £f* is an auxiliary variable 
of simulating is followed by a step of advancing said 55 value at iteration k, and d(j,i) is said delay between 
earliest simulation time of events yet to be simulated by changes occurring at subsystem J and their possible 
said processors. effect on subsystem i. 

3. The method of claim 1 wherein said step of simulat- 11. The method of claim 3 wherein said step of evalu- 
ing includes a step of evaluating whether a scheduled ating computes a simulated time value of a and said step 
event is to be simulated, based on changes occurring at 60 of simulating simulates events whose simulated time is 
a subset of said subsystems in response to previous simu- not greater than a, where a for subsystem i, a/, is the 
lation steps. value of afi in the final iteration that follows the steps: 

4. The method of claim 3 wherein the subsystems 
belonging to said subset are dependent on said prese- 
lected interval. 65 VSetaP^ +» "fiP = T r * = 0 

5. The method of claim 3 wherein said subset is con- 
trolled by said preselected interval and said delays be- 2: synchronize 
tween said subsystems. 
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-continued 

3: evaluate 1 = min 

yaieighborsO) {d{j,t) + nux{0/,Wfl}} 
Mi 

4: evaluate a!j+ x = nun{aA£* + x ) 
St synchronize 



4,901,260 



12 



6: evaluated < 



10 



min 1 ■ broadcast value of A to all nodes 



where T/ is the time of the event in subsystem i having 
the earliest simulation time, is an auxiliary variable 
value at iteration k,d(j,i) is said delay between changes 
occurring at subsystem j and their possible effect on 
subsystem i, and op// is the opaque period for subsystem 
j in the direction of subsystem i. 

12. The method of claim 4 wherein said step of evalu- 
ating computes a value of a and said step of simulating 



dated event list and the event lists of said neighbor- 
ing blocks, as required by said simulation of events; 
a synchronization processor associated with each of 
said blocks, connected to its associated event list 
and means for maintaining a, for developing values 
of said a; and 
means for interconnecting said plurality of event 
evaluation processors and synchronization proces- 
sors. 

16. The system of claim 16 wherein said synchroniza- 
tion processor develops a value of a based on values of 
Tj, from a preselected subset of said subsystems, where 
T; is the simulation time of the event in subsystem j 
having the earliest simulation time. 

17. The system of claim 16 where each of said syn- 
chronization processors develops a value of a for its 
block based on a consideration of potential changes in 
neighbors of its block. 

18. The system of claim 17 wherein said consideration 
20 is iterative. 

19. The system of claim 18 wherein said consideration 
takes into account opaque periods. 

20. The system of claim 16 where each of said syn- 
chronization processors develops a value of a for its 



15 



simulates events whose simulated time is not greater 25 block based 0 n a consideration of opaque periods of its 



than a, where a for subsystem i, a,*, is equal to 



rs(i) 



30 



where op/y is the opaque period for subsystem j in the 
direction of subsystem i. 

13. A system for performing discrete event simulation 
of a system having a plurality of interacting subsystems, 35 
comprising: 

a plurality of blocks, each block simulating a prese- 
lected subsystem and including an event list associ- 
ated with said preselected subsystem and means for 
maintaining information concerning the state of 40 
said preselected subsystem, and a, where a is an 
estimate of the earliest simulated time for which 
modification parameters can be modified by neigh- 
boring subsystems, where said modification param- 
eters include the information in said associated 45 
state registers and the events in said associated 
event; and 

a controller system for repetitively performing simu- 
lation of events in said event lists and re-evaluating 
the values of said a and ofTjioor* where Tjjoo/is the 50 
lowest event time of scheduled events to be simu- 
lated by said plurality of blocks. 

14. The system of claim 14 wherein said controller 
system comprises 

a plurality of interconnected controllers communicat- 55 
ing with said blocks, with each controller repeti- 
tively performing simulation of events in said event 
lists and re-evaluating the value of said a. 

15. The system of claim 14 wherein said controller 
system comprises 60 

an event evaluation processor associated with each of 
said blocks, connected to its associated event list, 
its associated means for maintaining state informa- 
tion, and to neighboring blocks, which are blocks 
that simulate subsystems that directly affect, or can 65 
be affected by, said preselected subsystem, where 
said event processor performs simulation of events 
in said associated event list and modifies said asso- 



neighbors. 

21. The system of claim 14 wherein said controller 
system comprises 

a plurality of controllers communicating with said 
blocks, with each controller repetitively perform- 
ing simulation of events in said event lists and re- 
evaluating the value of said a; and 

means for interconnecting said controllers. 

22. The system of claim 21 wherein said means for 
interconnecting receives a T, from each of said event 
lists, where Tj is the simulated time of the event to be 
simulated in said event list having the earliest simulation 
time, develops Tjhor which is min (T/), and returns 
T floor to each of said controllers. 

23. A system for performing discrete event simulation 
comprising: 

a plurality of blocks equal in number to the number of 
simulated interacting subsystems, each block simu- 
lating a preselected subsystem and including an 
event list associated with said preselected subsys- 
tem and registers for storing information concern- 
ing Tjioor* a, P and state information of said prese- 
lected subsystem; 

means for interconnecting said plurality of blocks; 

an event processing controller associated with each 
of said blocks, connected to its associated event list, 
its associated state registers, and to neighboring 
blocks, which are blocks that simulate subsystems 
that directly affect, or can be affected by, said 
preselected subsystem, where said event controller 
performs simulation of events in said associated 
event list and modifies said associated event list and 
the event lists of said neighboring blocks, as re- 
quired by said simulation of events; and 

a synchronization controller connected to its associ- 
ated event list, T/hon a, and & registers, and also 
connected to said means for interconnecting, for 
developing values for said T/hon a, and & registers, 
where Tjjoor is the earliest simulated time of an 
event to be simulated found in any of said event 
lists, a is an estimate of the earliest simulated time 
for which the information in said associated state 
registers or the events in said associated event list 
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can be modified by neighboring subsystems, and fi 
is an auxiliary estimate employed in evaluating a. 
24. A system for performing discrete event simulation 
comprising: 

a plurality of blocks equal in number to the number of 
simulated interacting subsystems, each block simu- 
lating a preselected subsystem and including an 
event list associated with said preselected subsys- 
tem and registers for storing information concern* 
ing T/joor* a* and state information of said prese- 
lected subsystem; 

means for interconnecting said plurality of blocks; 

an event processing controller associated with each 
of said blocks, connected to its associated event list, 
its associated state registers, and to a subset of said 
blocks, where said event controller performs simu- 



10 



15 



14 

lation of events in said associated event list and 
modifies said associated event list and the event 
lists of said neighboring blocks, as required by said 
simulation of events; and 
. synchronization controller connected to its associ- 
ated event list, T/joor and, a, registers, and also 
connected to said means for interconnecting, for 
developing values for said ^/loon and a, registers, 
where T/bor is the earliest simulated time of an 
event to be simulated found in any of said event 
lists, and a is an estimate of the earliest simulated 
time for which the information in said associated 
state registers or the events in said associated event 
list can be modified by neighboring subsystems. 



20 



25 



30 



35 



40 



45 



55 



65 



02/27/2004, EAST Version: 1.4.1 



